drop problem end characters from filename operating on String not RawFilePath

Fix bug that could cause an invalid utf-8 sequence to be used in a
temporary filename when the input filename was valid utf-8.

Sponsored-by: k0ld

diff --git a/CHANGELOG b/CHANGELOG
index 371f53f30f285e74722f189f800338d4d313cdf1..740253853a346ce896b1b01963b26906362c8912 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,6 +3,8 @@ git-annex (10.20250829) UNRELEASED; urgency=medium
   * drop: --fast support when dropping from a remote.
   * Fix crash operating on filenames that are exactly 21 bytes long
     and begin with a utf-8 character.
+  * Fix bug that could cause an invalid utf-8 sequence to be used in a
+    temporary filename when the input filename was valid utf-8.
   * git-annex.cabal: Turn on the OsPath build flag by default.
   * Add build warnings when git-annex is built without the OsPath
     build flag.

diff --git a/Utility/Tmp.hs b/Utility/Tmp.hs
index 582f6849fc898d6a361d7a452b5286cf51bb5107..df6673eadd78c33f1bb6d41f22ce25875dcdb5c2 100644 (file)
--- a/Utility/Tmp.hs
+++ b/Utility/Tmp.hs
@@ -116,20 +116,29 @@ relatedTemplate' :: RawFilePath -> RawFilePath
 #ifndef mingw32_HOST_OS
 relatedTemplate' f
        | len > templateAddedLength = 
-               {- Some filesystems like FAT have issues with filenames
-                - ending in ".", and others like VFAT don't allow a
-                - filename to end with trailing whitespace, so avoid
-                - truncating a filename to end that way. -}
-               let p = B.dropWhileEnd disallowed $
-                       truncateFilePath (len - templateAddedLength) f
+               let p = fixend $ truncateFilePath (len - templateAddedLength) f
                in if B.null p
                        then "t"
                        else p
        | otherwise = f
   where
        len = B.length f
-       disallowed c = c == dot || isSpace (chr (fromIntegral c))
+       {- Some filesystems like FAT have issues with filenames
+        - ending in ".", and others like VFAT don't allow a
+        - filename to end with trailing whitespace, so avoid
+        - truncating a filename to end that way. -}
+       fixend p =
+               {- B.dropWhileEnd doesn't take wide characters
+                - into account, but is fast, so use it to check
+                - the common case. -}
+               let p' = B.dropWhileEnd disallowed p
+               in if p' == p 
+                       then p
+                       else toRawFilePath $ reverse $
+                               dropWhile (disallowed . fromIntegral . ord) $
+                               reverse $ fromRawFilePath p
        dot = fromIntegral (ord '.')
+       disallowed c = c == dot || isSpace (chr (fromIntegral c))
 #else
 -- Avoids a test suite failure on windows, reason unknown, but
 -- best to keep paths short on windows anyway.

diff --git a/doc/bugs/multibyte_characters_broken.mdwn b/doc/bugs/multibyte_characters_broken.mdwn
index ff868a0419e17d08cea5695f1a71772982dbfd10..4b10d6419ff92bc20be98344c28946385cad5a46 100644 (file)
--- a/doc/bugs/multibyte_characters_broken.mdwn
+++ b/doc/bugs/multibyte_characters_broken.mdwn
@@ -31,3 +31,5 @@ The original file obviously has a correct encoding, but it seems that git annex
 
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 I use git annex to manage my whole music collection successfully.
+
+> [[fixed|done]] --[[Joey]]

diff --git a/doc/bugs/multibyte_characters_broken/comment_1_eb421648f585296f7c44f969bdcae7a4._comment b/doc/bugs/multibyte_characters_broken/comment_1_eb421648f585296f7c44f969bdcae7a4._comment
new file mode 100644 (file)
index 0000000..8660462
--- /dev/null
+++ b/doc/bugs/multibyte_characters_broken/comment_1_eb421648f585296f7c44f969bdcae7a4._comment
@@ -0,0 +1,28 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2025-09-15T16:59:59Z"
+ content="""
+git-annex actually attempts to truncate the filename taking unicode
+character width into account.
+
+Here is the truncation on the wrong byte though:
+
+       ghci> :t x
+       x :: String
+       ghci> x
+       "ingest-01-06 \19977\30707\29748\20035\12539\23500\27810\32654\26234\24693\12539\20037\24029\32190\12539\31712\21407\24693\32654\12539\28145\35211 \26792\21152 - Tuxedo Mirage.flac"
+       ghci> toRawFilePath x
+       "ingest-01-06 \228\184\137\231\159\179\231\144\180\228\185\131\227\131\187\229\175\140\230\178\162\231\190\142\230\153\186\230\129\181\227\131\187\228\185\133\229\183\157\231\182\190\227\131\187\231\175\160\229\142\159\230\129\181\231\190\142\227\131\187\230\183\177\232\166\139 \230\162\168\229\138\160 - Tuxedo Mirage.flac"
+       ghci> relatedTemplate (toRawFilePath x)
+       "ingest-01-06 \228\184\137\231\159\179\231\144\180\228\185\131\227\131\187\229\175\140\230\178\162\231\190\142\230\153\186\230\129\181\227\131\187\228\185\133\229\183\157\231\182\190\227\131\187\231\175\160\229\142\159\230\129\181\231\190\142\227\131\187\230\183\177\232\166\139 \230\162\168\229\138"
+
+What is going on is that '\160` is a space character, and filesystems like
+FAT do not allow a filename to end with a space. So relatedTemplate trims
+off trailing spaces, and accidentially trimmed off this byte, despite it
+being part of a multibyte sequence.
+
+Aren't filesystems with arbitrary limitations on what valid filenames are fun?
+
+Fixed this.
+"""]]